SuperNOVA: Semi-Parametric Identification and Estimation of Interaction and Effect Modification in Mixed Exposures using Stochastic Interventions in R.

Environmental epidemiology studies aim to understand the impact of mixed exposures on health outcomes while adjusting for covariates. However, traditional statistical methods make simplistic assumptions that may not be applicable to public policy decisions. Researchers are ultimately interested in answering causal questions, such as the impact of reducing toxic chemical exposures on adverse health outcomes like cancer. For example, in the case of PFAS, a class of chemicals measured simultaneously in blood samples, identifying the shifts that result in the greatest reduction in thyroid cancer rates can help more directly inform policy decisions on PFAS. In mixtures, nonlinear and non-additive relationships call for new statistical methods to estimate such modified exposure policies. To address these limitations, the open-source SuperNOVA package has been developed to use data-adaptive machine learning methods for identifying variable sets that have the most explanatory power on an outcome of interest. This package applies non-parametric definitions of interaction and effect modification to these variable sets in a mixed exposure, enabling researchers to explore modified treatment policies using stochastic interventions and answer causal questions. The SuperNOVA software implements the data-adaptive discovery of variable sets and estimation using optimal estimators for stochastic interventions described in our paper "Semi-Parametric Identification and Estimation of Interaction and Effect Modification in Mixed Exposures using Stochastic Interventions" (McCoy et al., 2023).

Comparing the effectiveness of a brief intervention to reduce unhealthy alcohol use among adult primary care patients with and without depression: A machine learning approach with augmented inverse probability weighting.

BACKGROUND: The combination of unhealthy alcohol use and depression is associated with adverse outcomes including higher rates of alcohol use disorder and poorer depression course. Therefore, addressing alcohol use among individuals with depression may have a substantial public health impact. We compared the effectiveness of a brief intervention (BI) for unhealthy alcohol use among patients with and without depression. METHOD: This observational study included 312,056 adult primary care patients at Kaiser Permanente Northern California who screened positive for unhealthy drinking between 2014 and 2017. Approximately half (48%) received a BI for alcohol use and 9% had depression. We examined 12-month changes in heavy drinking days in the previous three months, drinking days per week, drinks per drinking day, and drinks per week. Machine learning was used to estimate BI propensity, follow-up participation, and alcohol outcomes for an augmented inverse probability weighting (AIPW) estimator of the average treatment (BI) effect. This approach does not depend on the strong parametric assumptions of traditional logistic regression, making it more robust to model misspecification. RESULTS: BI had a significant effect on each alcohol use outcome in the non-depressed subgroup (-0.41 to -0.05, all ps < .003), but not in the depressed subgroup (-0.33 to -0.01, all ps > .28). However, differences between subgroups were nonsignificant (0.00 to 0.11, all ps > .44). CONCLUSION: On average, BI is an effective approach to reducing unhealthy drinking, but more research is necessary to understand its impact on patients with depression. AIPW with machine learning provides a robust method for comparing intervention effectiveness across subgroups.

Exploratory analysis of novel electronic health record variables for quantification of healthcare delivery strain, prediction of mortality, and prediction of imminent discharge.

OBJECTIVE: To explore the relationship between novel, time-varying predictors for healthcare delivery strain (eg, counts of patient orders per hour) and imminent discharge and in-hospital mortality. MATERIALS AND METHODS: We conducted a retrospective cohort study using data from adults hospitalized at 21 Kaiser Permanente Northern California hospitals between November 1, 2015 and October 31, 2020 and the nurses caring for them. Patient data extracted included demographics, diagnoses, severity measures, occupancy metrics, and process of care metrics (eg, counts of intravenous drip orders per hour). We linked these data to individual registered nurse records and created multiple dynamic, time-varying predictors (eg, mean acute severity of illness for all patients cared for by a nurse during a given hour). All analyses were stratified by patients' initial hospital unit (ward, stepdown unit, or intensive care unit). We used discrete-time hazard regression to assess the association between each novel time-varying predictor and the outcomes of discharge and mortality, separately. RESULTS: Our dataset consisted of 84 162 161 hourly records from 954 477 hospitalizations. Many novel time-varying predictors had strong associations with the 2 study outcomes. However, most of the predictors did not merely track patients' severity of illness; instead, many of them only had weak correlations with severity, often with complex relationships over time. DISCUSSION: Increasing availability of process of care data from automated electronic health records will permit better quantification of healthcare delivery strain. This could result in enhanced prediction of adverse outcomes and service delays. CONCLUSION: New conceptual models will be needed to use these new data elements.

Mixed Models for Repeated Measures Should Include Time-by-Covariate Interactions to Assure Power Gains and Robustness Against Dropout Bias Relative to Complete-Case ANCOVA.

In randomized trials with continuous-valued outcomes, the goal is often to estimate the difference in average outcomes between two treatment groups. However, the outcome in some trials is longitudinal, meaning that multiple measurements of the same outcome are taken over time for each subject. The target of inference in this case is often still the difference in averages at a given timepoint. One way to analyze these data is to ignore the measurements at intermediate timepoints and proceed with a standard covariate-adjusted analysis (e.g., ANCOVA) with the complete cases. However, it is generally thought that exploiting information from intermediate timepoints using mixed models for repeated measures (MMRM) (a) increases power and (b) more naturally "handles" missing data. Here, we prove that neither of these conclusions is entirely correct when baseline covariates are adjusted for without including time-by-covariate interactions. We back these claims up with simulations. MMRM provides benefits over complete-cases ANCOVA in many cases, but covariate-time interaction terms should always be included to guarantee the best results.

Increasing the efficiency of randomized trial estimates via linear adjustment for a prognostic score.

Estimating causal effects from randomized experiments is central to clinical research. Reducing the statistical uncertainty in these analyses is an important objective for statisticians. Registries, prior trials, and health records constitute a growing compendium of historical data on patients under standard-of-care that may be exploitable to this end. However, most methods for historical borrowing achieve reductions in variance by sacrificing strict type-I error rate control. Here, we propose a use of historical data that exploits linear covariate adjustment to improve the efficiency of trial analyses without incurring bias. Specifically, we train a prognostic model on the historical data, then estimate the treatment effect using a linear regression while adjusting for the trial subjects' predicted outcomes (their prognostic scores). We prove that, under certain conditions, this prognostic covariate adjustment procedure attains the minimum variance possible among a large class of estimators. When those conditions are not met, prognostic covariate adjustment is still more efficient than raw covariate adjustment and the gain in efficiency is proportional to a measure of the predictive accuracy of the prognostic model above and beyond the linear relationship with the raw covariates. We demonstrate the approach using simulations and a reanalysis of an Alzheimer's disease clinical trial and observe meaningful reductions in mean-squared error and the estimated variance. Lastly, we provide a simplified formula for asymptotic variance that enables power calculations that account for these gains. Sample size reductions between 10% and 30% are attainable when using prognostic models that explain a clinically realistic percentage of the outcome variance.

Evaluation of Incident 7-Day Infection and Sepsis Hospitalizations in an Integrated Health System.

Rationale: Prehospital opportunities to predict infection and sepsis hospitalization may exist, but little is known about their incidence following common healthcare encounters. Objectives: To evaluate the incidence and timing of infection and sepsis hospitalization within 7 days of living hospital discharge, emergency department discharge, and ambulatory visit settings. Methods: In each setting, we identified patients in clinical strata based on the presence of infection and severity of illness. We estimated number needed to evaluate values with hypothetical predictive model operating characteristics. Results: We identified 97,614,228 encounters, including 1,117,702 (1.1%) hospital discharges, 4,635,517 (4.7%) emergency department discharges, and 91,861,009 (94.1%) ambulatory visits between 2012 and 2017. The incidence of 7-day infection hospitalization varied from 37,140 (3.3%) following inpatient discharge to 50,315 (1.1%) following emergency department discharge and 277,034 (0.3%) following ambulatory visits. The incidence of 7-day infection hospitalization was increased for inpatient discharges with high readmission risk (10.0%), emergency department discharges with increased acute or chronic severity of illness (3.5% and 4.7%, respectively), and ambulatory visits with acute infection (0.7%). The timing of 7-day infection and sepsis hospitalizations differed across settings with an early rise following ambulatory visits, a later peak following emergency department discharges, and a delayed peak following inpatient discharge. Theoretical number needed to evaluate values varied by strata, but following hospital and emergency department discharge, were as low as 15-25. Conclusions: Incident 7-day infection and sepsis hospitalizations following encounters in routine healthcare settings were surprisingly common and may be amenable to clinical predictive models.

Unsupervised Learning for Automated Detection of Coronary Artery Disease Subgroups.

Background The promise of precision population health includes the ability to use robust patient data to tailor prevention and care to specific groups. Advanced analytics may allow for automated detection of clinically informative subgroups that account for clinical, genetic, and environmental variability. This study sought to evaluate whether unsupervised machine learning approaches could interpret heterogeneous and missing clinical data to discover clinically important coronary artery disease subgroups. Methods and Results The Genetic Determinants of Peripheral Arterial Disease study is a prospective cohort that includes individuals with newly diagnosed and/or symptomatic coronary artery disease. We applied generalized low rank modeling and K-means cluster analysis using 155 phenotypic and genetic variables from 1329 participants. Cox proportional hazard models were used to examine associations between clusters and major adverse cardiovascular and cerebrovascular events and all-cause mortality. We then compared performance of risk stratification based on clusters and the American College of Cardiology/American Heart Association pooled cohort equations. Unsupervised analysis identified 4 phenotypically and prognostically distinct clusters. All-cause mortality was highest in cluster 1 (oldest/most comorbid; 26%), whereas major adverse cardiovascular and cerebrovascular event rates were highest in cluster 2 (youngest/multiethnic; 41%). Cluster 4 (middle-aged/healthiest behaviors) experienced more incident major adverse cardiovascular and cerebrovascular events (30%) than cluster 3 (middle-aged/lowest medication adherence; 23%), despite apparently similar risk factor and lifestyle profiles. In comparison with the pooled cohort equations, cluster membership was more informative for risk assessment of myocardial infarction, stroke, and mortality. Conclusions Unsupervised clustering identified 4 unique coronary artery disease subgroups with distinct clinical trajectories. Flexible unsupervised machine learning algorithms offer the ability to meaningfully process heterogeneous patient data and provide sharper insights into disease characterization and risk assessment. Registration URL: https://www.clinicaltrials.gov; Unique identifier: NCT00380185.

Designing efficient randomized trials: power and sample size calculation when using semiparametric efficient estimators.

Trials enroll a large number of subjects in order to attain power, making them expensive and time-consuming. Sample size calculations are often performed with the assumption of an unadjusted analysis, even if the trial analysis plan specifies a more efficient estimator (e.g. ANCOVA). This leads to conservative estimates of required sample sizes and an opportunity for savings. Here we show that a relatively simple formula can be used to estimate the power of any two-arm, single-timepoint trial analyzed with a semiparametric efficient estimator, regardless of the domain of the outcome or kind of treatment effect (e.g. odds ratio, mean difference). Since an efficient estimator attains the minimum possible asymptotic variance, this allows for the design of trials that are as small as possible while still attaining design power and control of type I error. The required sample size calculation is parsimonious and requires the analyst to provide only a small number of population parameters. We verify in simulation that the large-sample properties of trials designed this way attain their nominal values. Lastly, we demonstrate how to use this formula in the "design" (and subsequent reanalysis) of a real randomized trial and show that fewer subjects are required to attain the same design power when a semiparametric efficient estimator is accounted for at the design stage.

Evaluation of an intervention targeted with predictive analytics to prevent readmissions in an integrated health system: observational study.

OBJECTIVES: To determine the associations between a care coordination intervention (the Transitions Program) targeted to patients after hospital discharge and 30 day readmission and mortality in a large, integrated healthcare system. DESIGN: Observational study. SETTING: 21 hospitals operated by Kaiser Permanente Northern California. PARTICIPANTS: 1 539 285 eligible index hospital admissions corresponding to 739 040 unique patients from June 2010 to December 2018. 411 507 patients were discharged post-implementation of the Transitions Program; 80 424 (19.5%) of these patients were at medium or high predicted risk and were assigned to receive the intervention after discharge. INTERVENTION: Patients admitted to hospital were automatically assigned to be followed by the Transitions Program in the 30 days post-discharge if their predicted risk of 30 day readmission or mortality was greater than 25% on the basis of electronic health record data. MAIN OUTCOME MEASURES: Non-elective hospital readmissions and all cause mortality in the 30 days after hospital discharge. RESULTS: Difference-in-differences estimates indicated that the intervention was associated with significantly reduced odds of 30 day non-elective readmission (adjusted odds ratio 0.91, 95% confidence interval 0.89 to 0.93; absolute risk reduction 95% confidence interval -2.5%, -3.1% to -2.0%) but not with the odds of 30 day post-discharge mortality (1.00, 0.95 to 1.04). Based on the regression discontinuity estimate, the association with readmission was of similar magnitude (absolute risk reduction -2.7%, -3.2% to -2.2%) among patients at medium risk near the risk threshold used for enrollment. However, the regression discontinuity estimate of the association with post-discharge mortality (-0.7% -1.4% to -0.0%) was significant and suggested benefit in this subgroup of patients. CONCLUSIONS: In an integrated health system, the implementation of a comprehensive readmissions prevention intervention was associated with a reduction in 30 day readmission rates. Moreover, there was no association with 30 day post-discharge mortality, except among medium risk patients, where some evidence for benefit was found. Altogether, the study provides evidence to suggest the effectiveness of readmission prevention interventions in community settings, but further research might be required to confirm the findings beyond this setting.

The Presentation, Pace, and Profile of Infection and Sepsis Patients Hospitalized Through the Emergency Department: An Exploratory Analysis.

To characterize the signs and symptoms of sepsis, compare them with those from simple infection and other emergent conditions and evaluate their association with hospital outcomes. DESIGN SETTING PARTICIPANTS AND INTERVENTION: A multicenter, retrospective cohort study of 408,377 patients hospitalized through the emergency department from 2012 to 2017 with sepsis, suspected infection, heart failure, or stroke. Infected patients were identified based on Sepsis-3 criteria, whereas noninfected patients were identified through diagnosis codes. MEASUREMENTS AND MAIN RESULTS: Signs and symptoms were identified within physician clinical documentation in the first 24 hours of hospitalization using natural language processing. The time of sign and symptom onset prior to presentation was quantified, and sign and symptom prevalence was assessed. Using multivariable logistic regression, the association of each sign and symptom with four outcomes was evaluated: sepsis versus suspected infection diagnosis, hospital mortality, ICU admission, and time of first antibiotics (> 3 vs ≤ 3 hr from presentation). A total of 10,825 signs and symptoms were identified in 6,148,348 clinical documentation fragments. The most common symptoms overall were as follows: dyspnea (35.2%), weakness (27.2%), altered mental status (24.3%), pain (23.9%), cough (19.7%), edema (17.8%), nausea (16.9%), hypertension (15.6%), fever (13.9%), and chest pain (12.1%). Compared with predominant signs and symptoms in heart failure and stroke, those present in infection were heterogeneous. Signs and symptoms indicative of neurologic dysfunction, significant respiratory conditions, and hypotension were strongly associated with sepsis diagnosis, hospital mortality, and intensive care. Fever, present in only a minority of patients, was associated with improved mortality (odds ratio, 0.67, 95% CI, 0.64-0.70; p < 0.001). For common symptoms, the peak time of symptom onset before sepsis was 2 days, except for altered mental status, which peaked at 1 day prior to presentation. CONCLUSIONS: The clinical presentation of sepsis was heterogeneous and occurred with rapid onset prior to hospital presentation. These findings have important implications for improving public education, clinical treatment, and quality measures of sepsis care.

Prediction of obstetrical and fetal complications using automated electronic health record data.

An increasing number of delivering women experience major morbidity and mortality. Limited work has been done on automated predictive models that could be used for prevention. Using only routinely collected obstetrical data, this study aimed to develop a predictive model suitable for real-time use with an electronic medical record. We used a retrospective cohort study design with split validation. The denominator consisted of women admitted to a delivery service. The numerator consisted of women who experienced a composite outcome that included both maternal (eg, uterine rupture, postpartum hemorrhage), fetal (eg, stillbirth), and neonatal (eg, hypoxic ischemic encephalopathy) adverse events. We employed machine learning methods, assessing model performance using the area under the receiver operator characteristic curve and number needed to evaluate. A total of 303,678 deliveries took place at 15 study hospitals between January 1, 2010, and March 31, 2018, and 4130 (1.36%) had ≥1 obstetrical complication. We employed data from 209,611 randomly selected deliveries (January 1, 2010, to March 31, 2017) as a derivation dataset and validated our findings on data from 52,398 randomly selected deliveries during the same time period (validation 1 dataset). We then applied our model to data from 41,669 deliveries from the last year of the study (April 1, 2017, to March 31, 2018 [validation 2 dataset]). Our model included 35 variables (eg, demographics, vital signs, laboratory tests, progress of labor indicators). In the validation 2 dataset, a gradient boosted model (area under the receiver operating characteristic curve or c statistic, 0.786) was slightly superior to a logistic regression model (c statistic, 0.778). Using an alert threshold of 4.1%, our final model would flag 16.7% of women and detect 52% of adverse outcomes, with a number needed to evaluate of 20.9 and 0.455 first alerts per day per 1000 annual deliveries. In conclusion, electronic medical record data can be used to predict obstetrical complications. The clinical utility of these automated models has not yet been demonstrated. To conduct interventions to assess whether using these models results in patient benefit, future work will need to focus on the development of clinical protocols suitable for use in interventions.

How Confident Are We about Observational Findings in Healthcare: A Benchmark Study.

Healthcare professionals increasingly rely on observational healthcare data, such as administrative claims and electronic health records, to estimate the causal effects of interventions. However, limited prior studies raise concerns about the real-world performance of the statistical and epidemiological methods that are used. We present the "OHDSI Methods Benchmark" that aims to evaluate the performance of effect estimation methods on real data. The benchmark comprises a gold standard, a set of metrics, and a set of open source software tools. The gold standard is a collection of real negative controls (drug-outcome pairs where no causal effect appears to exist) and synthetic positive controls (drug-outcome pairs that augment negative controls with simulated causal effects). We apply the benchmark using four large healthcare databases to evaluate methods commonly used in practice: the new-user cohort, self-controlled cohort, case-control, case-crossover, and self-controlled case series designs. The results confirm the concerns about these methods, showing that for most methods the operating characteristics deviate considerably from nominal levels. For example, in most contexts, only half of the 95% confidence intervals we calculated contain the corresponding true effect size. We previously developed an "empirical calibration" procedure to restore these characteristics and we also evaluate this procedure. While no one method dominates, self-controlled methods such as the empirically calibrated self-controlled case series perform well across a wide range of scenarios.

Automated Identification of Adults at Risk for In-Hospital Clinical Deterioration.

BACKGROUND: Hospitalized adults whose condition deteriorates while they are in wards (outside the intensive care unit [ICU]) have considerable morbidity and mortality. Early identification of patients at risk for clinical deterioration has relied on manually calculated scores. Outcomes after an automated detection of impending clinical deterioration have not been widely reported. METHODS: On the basis of a validated model that uses information from electronic medical records to identify hospitalized patients at high risk for clinical deterioration (which permits automated, real-time risk-score calculation), we developed an intervention program involving remote monitoring by nurses who reviewed records of patients who had been identified as being at high risk; results of this monitoring were then communicated to rapid-response teams at hospitals. We compared outcomes (including the primary outcome, mortality within 30 days after an alert) among hospitalized patients (excluding those in the ICU) whose condition reached the alert threshold at hospitals where the system was operational (intervention sites, where alerts led to a clinical response) with outcomes among patients at hospitals where the system had not yet been deployed (comparison sites, where a patient's condition would have triggered a clinical response after an alert had the system been operational). Multivariate analyses adjusted for demographic characteristics, severity of illness, and burden of coexisting conditions. RESULTS: The program was deployed in a staggered fashion at 19 hospitals between August 1, 2016, and February 28, 2019. We identified 548,838 non-ICU hospitalizations involving 326,816 patients. A total of 43,949 hospitalizations (involving 35,669 patients) involved a patient whose condition reached the alert threshold; 15,487 hospitalizations were included in the intervention cohort, and 28,462 hospitalizations in the comparison cohort. Mortality within 30 days after an alert was lower in the intervention cohort than in the comparison cohort (adjusted relative risk, 0.84, 95% confidence interval, 0.78 to 0.90; P<0.001). CONCLUSIONS: The use of an automated predictive model to identify high-risk patients for whom interventions by rapid-response teams could be implemented was associated with decreased mortality. (Funded by the Gordon and Betty Moore Foundation and others.).

Predicting preventable hospital readmissions with causal machine learning.

OBJECTIVE: To assess both the feasibility and potential impact of predicting preventable hospital readmissions using causal machine learning applied to data from the implementation of a readmissions prevention intervention (the Transitions Program). DATA SOURCES: Electronic health records maintained by Kaiser Permanente Northern California (KPNC). STUDY DESIGN: Retrospective causal forest analysis of postdischarge outcomes among KPNC inpatients. Using data from both before and after implementation, we apply causal forests to estimate individual-level treatment effects of the Transitions Program intervention on 30-day readmission. These estimates are used to characterize treatment effect heterogeneity and to assess the notional impacts of alternative targeting strategies in terms of the number of readmissions prevented. DATA COLLECTION: 1 539 285 index hospitalizations meeting the inclusion criteria and occurring between June 2010 and December 2018 at 21 KPNC hospitals. PRINCIPAL FINDINGS: There appears to be substantial heterogeneity in patients' responses to the intervention (omnibus test for heterogeneity p = 2.23 × 10-7 ), particularly across levels of predicted risk. Notably, predicted treatment effects become more positive as predicted risk increases; patients at somewhat lower risk appear to have the largest predicted effects. Moreover, these estimates appear to be well calibrated, yielding the same estimate of annual readmissions prevented in the actual treatment subgroup (1246, 95% confidence interval [CI] 1110-1381) as did a formal evaluation of the Transitions Program (1210, 95% CI 990-1430). Estimates of the impacts of alternative targeting strategies suggest that as many as 4458 (95% CI 3925-4990) readmissions could be prevented annually, while decreasing the number needed to treat from 33 to 23, by targeting patients with the largest predicted effects rather than those at highest risk. CONCLUSIONS: Causal machine learning can be used to identify preventable hospital readmissions, if the requisite interventional data are available. Moreover, our results suggest a mismatch between risk and treatment effects.

Assessment of Value of Neighborhood Socioeconomic Status in Models That Use Electronic Health Record Data to Predict Health Care Use Rates and Mortality.

Importance: Prediction models are widely used in health care as a way of risk stratifying populations for targeted intervention. Most risk stratification has been done using a small number of predictors from insurance claims. However, the utility of diverse nonclinical predictors, such as neighborhood socioeconomic contexts, remains unknown. Objective: To assess the value of using neighborhood socioeconomic predictors in the context of 1-year risk prediction for mortality and 6 different health care use outcomes in a large integrated care system. Design, Setting, and Participants: Diagnostic study using data from all adults age 18 years or older who had Kaiser Foundation Health Plan membership and/or use in the Kaiser Permantente Northern California: a multisite, integrated health care delivery system between January 1, 2013, and June 30, 2014. Data were recorded before the index date for each patient to predict their use and mortality in a 1-year post period using a test-train split for model training and evaluation. Analyses were conducted in fall of 2019. Main Outcomes and Measures: One-year encounter counts (doctor office, virtual, emergency department, elective hospitalizations, and nonelective), total costs, and mortality. Results: A total of 2 951 588 patients met inclusion criteria (mean [SD] age, 47.2 [17.4] years; 47.8% were female). The mean (SD) Neighborhood Deprivation Index was -0.32 (0.84). The areas under the receiver operator curve ranged from 0.71 for emergency department use (using the LASSO method and electronic health record predictors) to 0.94 for mortality (using the random forest method and electronic health record predictors). Neighborhood socioeconomic status predictors did not meaningfully increase the predictive performance of the models for any outcome. Conclusions and Relevance: In this study, neighborhood socioeconomic predictors did not improve risk estimates compared with what is obtainable using standard claims data regardless of model used.

Association Between Peripheral Blood Oxygen Saturation (SpO2)/Fraction of Inspired Oxygen (FiO2) Ratio Time at Risk and Hospital Mortality in Mechanically Ventilated Patients.

INTRODUCTION: Acute respiratory failure requiring mechanical ventilation is a leading cause of mortality in the intensive care unit. Although single peripheral blood oxygen saturation/fraction of inspired oxygen (SpO2/FiO2) ratios of hypoxemia have been evaluated to risk-stratify patients with acute respiratory distress syndrome, the utility of longitudinal SpO2/FiO2 ratios is unknown. OBJECTIVE: To assess time-based SpO2/FiO2 ratios ≤ 150-SpO2/FiO2 time at risk (SF-TAR)-for predicting mortality in mechanically ventilated patients. METHODS: Retrospective, observational cohort study of mechanically ventilated patients at 21 community and 2 academic hospitals. Association between the SF-TAR in the first 24 hours of ventilation and mortality was examined using multivariable logistic regression and compared with the worst recorded isolated partial pressure of arterial oxygen/fraction of inspired oxygen (P/F) ratio. RESULTS: In 28,758 derivation cohort admissions, every 10% increase in SF-TAR was associated with a 24% increase in adjusted odds of hospital mortality (adjusted odds ratio = 1.24; 95% confidence interval [CI] = 1.23-1.26); a similar association was observed in validation cohorts. Discrimination for mortality modestly improved with SF-TAR (area under the receiver operating characteristic curve [AUROC] = 0.81; 95% CI = 0.81-0.82) vs the worst P/F ratio (AUROC = 0.78; 95% CI = 0.78-0.79) and worst SpO2/FiO2 ratio (AUROC = 0.79; 95% CI = 0.79-0.80). The SF-TAR in the first 6 hours offered comparable discrimination for hospital mortality (AUROC = 0.80; 95% CI = 0.79-0.80) to the 24-hour SF-TAR. CONCLUSION: The SF-TAR can identify ventilated patients at increased risk of death, offering modest improvements compared with single SpO2/FiO2 and P/F ratios. This longitudinal, noninvasive, and broadly generalizable tool may have particular utility for early phenotyping and risk stratification using electronic health record data in ventilated patients.

An evaluation of clinical order patterns machine-learned from clinician cohorts stratified by patient mortality outcomes.

OBJECTIVE: Evaluate the quality of clinical order practice patterns machine-learned from clinician cohorts stratified by patient mortality outcomes. MATERIALS AND METHODS: Inpatient electronic health records from 2010 to 2013 were extracted from a tertiary academic hospital. Clinicians (n = 1822) were stratified into low-mortality (21.8%, n = 397) and high-mortality (6.0%, n = 110) extremes using a two-sided P-value score quantifying deviation of observed vs. expected 30-day patient mortality rates. Three patient cohorts were assembled: patients seen by low-mortality clinicians, high-mortality clinicians, and an unfiltered crowd of all clinicians (n = 1046, 1046, and 5230 post-propensity score matching, respectively). Predicted order lists were automatically generated from recommender system algorithms trained on each patient cohort and evaluated against (i) real-world practice patterns reflected in patient cases with better-than-expected mortality outcomes and (ii) reference standards derived from clinical practice guidelines. RESULTS: Across six common admission diagnoses, order lists learned from the crowd demonstrated the greatest alignment with guideline references (AUROC range = 0.86-0.91), performing on par or better than those learned from low-mortality clinicians (0.79-0.84, P < 10-5) or manually-authored hospital order sets (0.65-0.77, P < 10-3). The same trend was observed in evaluating model predictions against better-than-expected patient cases, with the crowd model (AUROC mean = 0.91) outperforming the low-mortality model (0.87, P < 10-16) and order set benchmarks (0.78, P < 10-35). DISCUSSION: Whether machine-learning models are trained on all clinicians or a subset of experts illustrates a bias-variance tradeoff in data usage. Defining robust metrics to assess quality based on internal (e.g. practice patterns from better-than-expected patient cases) or external reference standards (e.g. clinical practice guidelines) is critical to assess decision support content. CONCLUSION: Learning relevant decision support content from all clinicians is as, if not more, robust than learning from a select subgroup of clinicians favored by patient outcomes.

Inpatient Clinical Order Patterns Machine-Learned From Teaching Versus Attending-Only Medical Services.

Clinical order patterns derived from data-mining electronic health records can be a valuable source of decision support content. However, the quality of crowdsourcing such patterns may be suspect depending on the population learned from. For example, it is unclear whether learning inpatient practice patterns from a university teaching service, characterized by physician-trainee teams with an emphasis on medical education, will be of variable quality versus an attending-only medical service that focuses strictly on clinical care. Machine learning clinical order patterns by association rule episode mining from teaching versus attending-only inpatient medical services illustrated some practice variability, but converged towards similar top results in either case. We further validated the automatically generated content by confirming alignment with external reference standards extracted from clinical practice guidelines.